29 research outputs found
Training of Convolutional Networks on Multiple Heterogeneous Datasets for Street Scene Semantic Segmentation
We propose a convolutional network with hierarchical classifiers for
per-pixel semantic segmentation, which is able to be trained on multiple,
heterogeneous datasets and exploit their semantic hierarchy. Our network is the
first to be simultaneously trained on three different datasets from the
intelligent vehicles domain, i.e. Cityscapes, GTSDB and Mapillary Vistas, and
is able to handle different semantic level-of-detail, class imbalances, and
different annotation types, i.e. dense per-pixel and sparse bounding-box
labels. We assess our hierarchical approach, by comparing against flat,
non-hierarchical classifiers and we show improvements in mean pixel accuracy of
13.0% for Cityscapes classes and 2.4% for Vistas classes and 32.3% for GTSDB
classes. Our implementation achieves inference rates of 17 fps at a resolution
of 520x706 for 108 classes running on a GPU.Comment: IEEE Intelligent Vehicles 201
Towards holistic scene understanding:Semantic segmentation and beyond
This dissertation addresses visual scene understanding and enhances
segmentation performance and generalization, training efficiency of networks,
and holistic understanding. First, we investigate semantic segmentation in the
context of street scenes and train semantic segmentation networks on
combinations of various datasets. In Chapter 2 we design a framework of
hierarchical classifiers over a single convolutional backbone, and train it
end-to-end on a combination of pixel-labeled datasets, improving
generalizability and the number of recognizable semantic concepts. Chapter 3
focuses on enriching semantic segmentation with weak supervision and proposes a
weakly-supervised algorithm for training with bounding box-level and
image-level supervision instead of only with per-pixel supervision. The memory
and computational load challenges that arise from simultaneous training on
multiple datasets are addressed in Chapter 4. We propose two methodologies for
selecting informative and diverse samples from datasets with weak supervision
to reduce our networks' ecological footprint without sacrificing performance.
Motivated by memory and computation efficiency requirements, in Chapter 5, we
rethink simultaneous training on heterogeneous datasets and propose a universal
semantic segmentation framework. This framework achieves consistent increases
in performance metrics and semantic knowledgeability by exploiting various
scene understanding datasets. Chapter 6 introduces the novel task of part-aware
panoptic segmentation, which extends our reasoning towards holistic scene
understanding. This task combines scene and parts-level semantics with
instance-level object detection. In conclusion, our contributions span over
convolutional network architectures, weakly-supervised learning, part and
panoptic segmentation, paving the way towards a holistic, rich, and sustainable
visual scene understanding.Comment: PhD Thesis, Eindhoven University of Technology, October 202
A Domain Agnostic Normalization Layer for Unsupervised Adversarial Domain Adaptation
We propose a normalization layer for unsupervised domain adaption in semantic
scene segmentation. Normalization layers are known to improve convergence and
generalization and are part of many state-of-the-art fully-convolutional neural
networks. We show that conventional normalization layers worsen the performance
of current Unsupervised Adversarial Domain Adaption (UADA), which is a method
to improve network performance on unlabeled datasets and the focus of our
research. Therefore, we propose a novel Domain Agnostic Normalization layer and
thereby unlock the benefits of normalization layers for unsupervised
adversarial domain adaptation. In our evaluation, we adapt from the synthetic
GTA5 data set to the real Cityscapes data set, a common benchmark experiment,
and surpass the state-of-the-art. As our normalization layer is domain agnostic
at test time, we furthermore demonstrate that UADA using Domain Agnostic
Normalization improves performance on unseen domains, specifically on
Apolloscape and Mapillary
On Boosting Semantic Street Scene Segmentation with Weak Supervision
Training convolutional networks for semantic segmentation requires per-pixel
ground truth labels, which are very time consuming and hence costly to obtain.
Therefore, in this work, we research and develop a hierarchical deep network
architecture and the corresponding loss for semantic segmentation that can be
trained from weak supervision, such as bounding boxes or image level labels, as
well as from strong per-pixel supervision. We demonstrate that the hierarchical
structure and the simultaneous training on strong (per-pixel) and weak
(bounding boxes) labels, even from separate datasets, constantly increases the
performance against per-pixel only training. Moreover, we explore the more
challenging case of adding weak image-level labels. We collect street scene
images and weak labels from the immense Open Images dataset to generate the
OpenScapes dataset, and we use this novel dataset to increase segmentation
performance on two established per-pixel labeled datasets, Cityscapes and
Vistas. We report performance gains up to +13.2% mIoU on crucial street scene
classes, and inference speed of 20 fps on a Titan V GPU for Cityscapes at 512 x
1024 resolution. Our network and OpenScapes dataset are shared with the
research community.Comment: Oral presentation IEEE IV 201
Panoptic Segmentation with a Joint Semantic and Instance Segmentation Network
We present a single network method for panoptic segmentation. This method
combines the predictions from a jointly trained semantic and instance
segmentation network using heuristics. Joint training is the first step towards
an end-to-end panoptic segmentation network and is faster and more memory
efficient than training and predicting with two networks, as done in previous
work. The architecture consists of a ResNet-50 feature extractor shared by the
semantic segmentation and instance segmentation branch. For instance
segmentation, a Mask R-CNN type of architecture is used, while the semantic
segmentation branch is augmented with a Pyramid Pooling Module. Results for
this method are submitted to the COCO and Mapillary Joint Recognition Challenge
2018. Our approach achieves a PQ score of 17.6 on the Mapillary Vistas
validation set and 27.2 on the COCO test-dev set.Comment: Technical repor
Part-aware Panoptic Segmentation
In this work, we introduce the new scene understanding task of Part-aware
Panoptic Segmentation (PPS), which aims to understand a scene at multiple
levels of abstraction, and unifies the tasks of scene parsing and part parsing.
For this novel task, we provide consistent annotations on two commonly used
datasets: Cityscapes and Pascal VOC. Moreover, we present a single metric to
evaluate PPS, called Part-aware Panoptic Quality (PartPQ). For this new task,
using the metric and annotations, we set multiple baselines by merging results
of existing state-of-the-art methods for panoptic segmentation and part
segmentation. Finally, we conduct several experiments that evaluate the
importance of the different levels of abstraction in this single task.Comment: CVPR 2021. Code and data: https://github.com/tue-mps/panoptic_part
Antiretroviral activity of 5-azacytidine during treatment of a HTLV-1 positive myelodysplastic syndrome with autoimmune manifestations
Myelodysplastic syndromes (MDS) are often accompanied by autoimmune phenomena. The underlying mechanisms for these associations remain uncertain, although T cell activation seems to be important. Human T-lymphotropic virus (HTLV-1) has been detected in patients with myelodysplastic syndromes, mostly in regions of the world which are endemic for the virus, and where association of HTLV-1 with rheumatological manifestation is not rare. We present here the case of a 58 year old man who presented with cytopenias, leukocytoclastic vasculitis of the skin and glomerulopathy, and was diagnosed as MDS (refractory anemia with excess blasts - RAEB 1). The patient also tested positive for HTLV-1 by PCR. After 8 monthly cycles of 5-azacytidine he achieved a complete hematologic remission. Following treatment, a second PCR for HTLV-1 was carried out and found to be negative. This is the first report in the literature of a HTLV-1-positive MDS with severe autoimmune manifestations, which was treated with the hypomethylating factor 5-azacitidine, achieving cytogenetic remission with concomitant resolution of the autoimmune manifestations, as well as HTLV-1-PCR negativity. HTLV-1-PCR negativity may be due to either immune mediated clearance of the virus, or a potential antiretroviral effect of 5-azacytidine. 5-azacytidine is known for its antiretroviral effects, although there is no proof of its activity against HTLV-1 infection in vivo